A Framework for Analyzing Linux System Overheads on HPC Applications∗

Linux currently plays an important role in high-end computing systems, but recent work has shown that Linux-related processing costs and variablity in network processing times can limit the scalability of HPC applications. Measuring and understanding these overheads is thus key for future use of Linux in large scale HPC systems. Unfortunately, currently available performance monitoring systems introduce large overheads, performance data is generally not available on-line or from the operating system, and the data collected by such systems is generally coarse-grained. In this paper, we present a low-overhead framework for solving one of these problems: making useful operating system performance data available to the application at runtime. Specifically, we have enhanced Linux Trace Toolkit(LTT) to monitor the performance characteristics of individual system calls and to make per-request performance data available to the application. We demonstrate the ability of this framework to monitor individual network and disk requests, and show that the overhead of our per-request performance monitoring framework is minimal. We also present preliminary measurements of Linux system call overhead on a simple HPC.